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IMAGE PICKUP DEVICE AND IMAGE PICKUP METHOD 



CROSS-REFERENCE TO RELATED APPLICATIONS 



The present application claims priority under 35 U.S.C. §1 19 of Japanese 



Application No. 2004-342457, filed on November 26, 2004, and Japanese Patent Application 
No. 2005-108726, filed on April 5, 2005, the entire content of which is expressly incorporated by 
reference herein. 



The present invention pertains to an image pickup device and an image pickup 
method, in particular to an image pickup device and image pickup method that obtains images of 
the periphery of a vehicle and displays them to the driver of the vehicle. 



Periphery visual confirmation devices for vehicles in which cameras are placed to the 
front and the rear of a vehicle to take images of the field of vision on both sides of the vehicle 
and display images of blind spots in the periphery of the vehicle onto a display are known as 
conventional technology. (For example, refer to Japanese Unexamined Patent Application 
Publication No. 3468661). 

In the '661 application, not only is technology disclosed in which the aforementioned 
function is enabled via operation of a switch, but technology is also disclosed whereby images of 
side views of the field of vision are displayed in accordance with the environment in which the 
vehicle is traveling as indicated by driving history and navigation information. 



However, in (Japanese Application No. 3468661), since the range in which the 
camera can pickup images is limited, the best images of blind spots cannot be displayed in 
accordance with the conditions in which the vehicle enters the road. For example, if a vehicle 
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enters the road diagonally, the best range of observation cannot be displayed. In addition, even if 
multiple cameras are installed, tiie best range of observation still may not be displayed. 

The present invention is characterized in that the main point is to provide an image 
pickup device provided with a plurality of periphery image pickup means for obtaining images of 
the periphery of a vehicle, wherein said periphery image pickup means that obtain images to 
display for the driver of the vehicle and the image range of the images obtained by the periphery 
image pickup means that are displayed to the driver of the vehicle are selected in accordance 
with the conditions in which the vehicle enters a road that intersects the direction in which the 
vehicle is traveling. 

According to the present invention, an image pickup device and image pickup method 
are provided for displaying images of the best range of observation without relying on the 
conditions in which a vehicle enters the road or on the installation of multiple cameras. 

BRIEF DESCRIPTION OF DRAWINGS 
Figure 1 is a block diagram showing the image pickup device pertaining to Embodiment 

1 of the present invention; 

Figure 2 is a flow chart showing the process for the image pickup device shown in Figure 

1; 

Figure 3 is a block diagram showing the image pickup device pertaining to Embodiment 

2 of the present invention; 

Figure 4 is a flow chart showing the process for the image pickup device shown in Figure 

3; 

Figure 5 is a block diagram showing the image pickup device pertaining to Embodiment 

3 of the present invention; 

Figure 6 is a flow chart showing the process for the image pickup device shown in Figure 

5; 
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Figure 7(a) is a plan view showing an illustration of a vehicle provided with the image 
pickup device shown in Figurel in which the vehicle is entering the road at a diagonal in a 
forward-moving direction; 

Figure 7(b) is a plan view of an illustration of a vehicle provided with the image pickup 
device shown in Figure 3 in which the vehicle is entering the road at a diagonal in a forward- 
moving direction; 

Figure 7(c) is a plan view showing an illustration of a vehicle provided with the image 
pickup device shown in Figure 5 in which the vehicle is entering the road at a diagonal in a 
reverse direction; 

Figures 8 is a plan view of the illustration shown in Figure 7(a) showing the arrangement 
of the wide-angle cameras and their image pickup range; 

Figure 9 is a plan view of the illustrations shown in Figure 7(b) and (c) showing the 
arrangement of the wide-angle cameras and their image pickup range; 

Figures 10 is a plan view of a vehicle equipped with the wide-angle cameras shown in 
Figure 9 in which the vehicle is starting to enter the road in a forward-moving direction from a 
parking lot or a narrow road; 

Figures 11 is a plan view of a vehicle that is advancing even further onto the road than 
that shown in Figure 10 in which the vehicle is starting to turn left; 

Figure 12(a) shows the image displayed on the display monitor provided on the image 
pickup device pertaining to the comparative example (Example 1); 

Figure 12(b) is a plan view of the entry conditions of a vehicle when it takes the image 
shown in Figure 12(a) (Example 1); 

Figure 13(a) shows the image displayed on the display monitor provided on the image 
pickup device pertaining to the comparative example (Example 2); 

Figure 13(b) is a plan view of the entry conditions of a vehicle when it takes the image 
shown in Figure 13(a) (Example 2); 

Figure 14 is a block diagram of the overall configuration of the image pickup device 
pertaining to Embodiment 4 of the present invention; 

Figure 15(a) is one example of the camera position and image pickup ranges for each 
camera installed on the vehicle, as shown in Figure 14; 
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Figure 15(b) is a graph explaining the basis for defining the image pickup range; 

Figure 16 is one example of the situation in which the image pickup device shown in 
Figure 14 operates (Example 1). In this Figure, (a) is a road diagram showing the position of the 
vehicle and the road and (b) is a type diagram showing the base line and target range set by the 
image pickup device; 

Figure 17 is another example of the situation in which the image pickup device shown in 
FigureH operates (Example 2). In this figure, (a) is a road diagram showing the position of the 
vehicle and the road and (b) is a type diagram showing the base line and target range set by the 
image pickup device; 

Figure 18 is another example of the situation in which the image pickup device shown in 
FigureH operates (Example 3). In this figure, (a) is a road diagram showing the position of the 
vehicle and the road and (b) is a type diagram showing the base line and target range set by the 
image pickup device; 

Figure 19 is an explanation of the method used for prioritizing the cameras when the 
image pickup device in FigureH selects the camera images, which corresponds to the situation 
shown in Figure 1 8; 

Figure 20 is a type diagram showing the display screen for the display monitor shown in 
FigureH; 

Figure 21 is an explanation of the display policy; and 

Figure 22 is a flow chart showing the image pickup method employed by the image 
pickup device shown in FigureH. 

DETAILED DESCRIPTION 

Below is provided an explanation of the embodiments for executing the present invention 
with reference to the Figures. All parts in the Figures that are the same or similar have been 
referenced using the same symbols. 

A device that has already been marketed is shown in Figure 12(b), in which cameras 60 
and 61 are arranged on either side of the fi-ont of vehicle 24 in order to observe blind spots 72a 
and 72b at intersections with poor visibility. This Figure illustrates an example of an intersection 
with poor visibility in which the vehicle enters a narrow road surrounded on both sides by high 
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walls 31a and 31b from a wide road. As shown in Figure 12(b), when the vehicle approaches 
from a wide road in a perpendicular direction, cameras 60 and 61 can observe the correct blind 
spots 72a and 72b. In other words, as shown in Figure 12(a), person 51 that is in the correct blind 
spot 72b is projected in image 52b, which is indicated to the driver of the vehicle so that the 
driver can be aware of the person's presence. 

However, since range 72a and 72b, in which cameras 60 and 61 can pickup images is 
limited, if image pickup range 72a and 72b do not coincide with the correct blind spots, the 
driver cannot observe the blind spots via the images picked up by cameras 60 and 61 . For 
example, as shown in Figure 13(b), if a vehicle approaches from a wide road diagonally instead 
of perpendicularly, cameras 60 and 61 cannot observe correct blind spots 72a and 72b. As shown 
in Figure 1 3(a), only wall 3 1 a is projected in image 52a that is picked up by left camera 60 and 
the correct blind spot is not projected in image 52b that is picked up by right camera 61, so the 
driver is not aware of person 51 that is in the blind spot. Generally speaking, since there are 
various different types of entry conditions for a vehicle, entry conditions in which the necessary 
observations can be made are actually very limited. 

As shown in Figure 1 , the image pickup device pertaining to Embodiment 1 for executing 
the present invention provides periphery image pickup portion 1 comprised of a plurality of 
periphery image pickup means (a plurality of cameras) that obtain images of the periphery of a 
vehicle wherein the camera that obtains the image displayed to the driver of the vehicle and the 
image range of the image obtained by said camera and displayed to the driver of the vehicle are 
selected according to the conditions in which said vehicle enters a road that intersects the 
direction in which the vehicle is traveling. 

More specifically, the image pickup device in Figure 1 provides a plurality of cameras 10 
and 1 1 for pickup images of the periphery of the vehicle, starting switch 12 as one example of a 
starting point detecting means that detects the starting point at which images picked up by said 
plurality of cameras 10 and 1 1 begin to get displayed to the driver of the vehicle, entry 
conditions detecting portion 3 as one example of an entry conditions detecting means that detects 
the conditions in which a vehicle enters the road at the display starting point detected by starting 
switch 12, image range adjusting portion 15 as one example of an image range adjustment means 
that adjusts the image range displayed in accordance with the entry conditions detected by entry 
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conditions detecting portion 3 and display monitor 16 as one example of a display means that 
displays the image range that is adjusted by image range adjusting portion 15. 

The term, "entry conditions", as used herein, refers to the entry angle and position of the 
vehicle in relation to the extended direction (lengthwise) of the road that intersects with the 
direction in which the vehicle is traveling. Display monitor 16 appropriately displays the range 
of observation based on the conditions in which the vehicle enters the road. The plurality of 
cameras that constitute periphery image pickup portion 1 consist of side cameras (left wide-angle 
camera 10 and right wide-angle camera 1 1) arranged at either side of the vehicle. Left wide- 
angle camera 10 and right wide-angle camera 1 1 can each obtain images at a 180-degree wide- 
angle range. In addition, included in entry conditions detecting portion 3 are navigation 1 3 for 
obtaining information on the position of the vehicle and the surrounding roads and gyro 14 for 
obtaining absolute direction information for the vehicle. 

As shown in Figure 8, left wide-angle camera 10 and right wide-angle camera 1 1 are each 
arranged at the front end of vehicle 24 to obtain 1 80-degree wide-angle range images 22a and 
22b. In Figure 7(a), is an illustration of the vehicle equipped with the image pickup device shown 
in Figure 1 that is diagonally entering the side on which the road is located in a forward-moving 
direction from the side on which the vehicle is parked. In other words, traveling direction 21 
indicates a diagonal entry condition and not a perpendicular direction in relation to the road. So, 
when the vehicle enters the road from a parking lot in a diagonal direction, image range adjusting 
portion 15 adjusts the portion that corresponds to the correct blind spot range from image pickup 
range 22a, 22b taken by left wide-angle camera 10 and right wide-angle camera 1 1 to create 
adjusted display range 23a and 23b, which are displayed on display monitor 16. In other words, 
the range of the picture angle for the side views can be adjusted separately for both right and left 
depending on the entry conditions. 

In this manner, as shown in Figure 7(a), for example, when entering the road from a 
parking lot or when entering the highway from a narrow road, the left and right images of the 
road at the point at which only the front end of the vehicle enters the road are projected onto 
display monitor 16 located inside the vehicle and the traffic situation on the road can be 
confirmed. 
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In addition, after the vehicle enters the road from a parking lot or a narrow road, or the 
like, and proceeds to make a left turn or when the angle of entry in relation to the road is already 
at an angle, as shown in Figure 7(a), the picture angle range required to confirm the right and left 
road situation from the image range obtained by the wide-angle camera in accordance with the 
vehicle's entry conditions (the entry direction, angle and position of the vehicle in relation to the 
lengthwise direction of the road) can be extracted by means of image conversion in order to 
provide the appropriate image range. 

Thus, even when the vehicle is turning as it enters the road or when the angle of entry is 
not at a right angle, the situation of the left and right sides of the road can be displayed onto 
display monitor 16. By displaying such an image, particularly while a vehicle is turning, the 
presence of other vehicles can be confirmed in situations such as when the driver of a vehicle 
fails to see the other vehicle when conducting a safety check of the road ahead or when another 
vehicle approaches at a speed that exceeds the legal speed limit, thus allowing for a safer entry 
onto the road. 

At this point, the entry conditions (angle and position) for the vehicle entering the road 
can also be calculated from road information obtained by navigation 13 and information obtained 
by gyro 14. 

In addition, it is also desirable to pre-set the optimum picture angle range in order to 
confirai the situation of the left and right sides of the road to determine the entry conditions for 
the vehicle entering the road. This will allow for an increasingly appropriate image range. 
Furthermore, if the picture angle range is stored in navigation 13 in accordance with the width of 
the road onto which the vehicle is entering, the maximum setting can be achieved for all types of 
roads. 

Next, an explanation is provided of the process for the image pickup device shown in 
Figure 1 with reference to the flow chart in Figure 2. 

(a) First at Step SI 01, the ignition is turned ON at the discretion of the driver. At Step 
SI 02, the process determines whether the ignition has been tumed OFF or not by the driver. If 
the ignition is ON (is ON at Step SI 02), the process proceeds to SI 03 and if the ignition is OFF 
(is OFF at Step SI 02), the flow process ends. 
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(b) Next, at Step SI 03, the process determines the start of the image pickup device shown 
in Figure 1 . If the image pickup device has been started (if it is ON at Step SI 03), the process 
moves to Step SI 04 and if the image pickup device has not been started (if it is OFF at Step 

S 1 03), the process returns to Step S 1 02. 

(c) Next, at Step SI 04, image range adjusting portion 15 obtains the image signal 
forwarded from wide-angle cameras 10 and 1 1 and at Step SI 05, image range adjusting portion 
15 obtains the current position of the vehicle and the map information of the vicinity from 
navigation 13. At Step SI 06, image range adjusting portion 15 obtains the absolute direction 
information of the vehicle from gyro 14. 

(d) Next, at Step SI 07, the current position of the vehicle and the map information of the 
vicinity from navigation 13 and the absolute direction information of the vehicle from gyro 14 
are used to calculate the direction of entry onto the road (forward or reverse direction/angle of 
entry (the angle in relation to the road)/entry position (the distance at which the vehicle advances 
onto the road). The entry position is defined as the distance from the base line of the road (for 
example, the line on the side at which the vehicle enters within the line that marks the width of 
the road) to the base line of the vehicle (for example the center of the rear wheel axle). At Step 
SI 08, the position (absolute position)/direction (absolute direction) of the camera installed on the 
vehicle is specified. 

(e) Next, at Step SI 09, the range required for observation is specified in accordance with 
the entry conditions and at Step SI 10, image range adjusting portion 15 uses the image 
conversion to adjust the image range. Finally, at Step Sill, display monitor 16 displays the 
adjusted image to the driver. After this, the process returns to SI 02, and Steps S102-S1 1 1 are 
repeated to perform the process again. 

It is desirable to execute Steps SI 02-81 1 1 while the vehicle is in the process of entering 
the road as well as after the vehicle has entered the road. In other words, even after the vehicle 
has entered the road, it is desirable to select cameras 10 and 11, which have obtained the images 
displayed to the driver of the vehicle and the image range of the images obtained by cameras 10 
and 1 1 that is displayed to the driver of the vehicle in accordance with the conditions in which 
the vehicle enters the road. 
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As explained above, since the image pickup device pertaining to Embodiment 1 for 
executing the present invention is provided with entry conditions detecting portion 3 that detects 
the conditions in which a vehicle enters the road at the display starting point and image range 
adjusting portion 15 that adjusts the image range displayed in accordance with the conditions in 
which the vehicle enters the road, the optimum range of observation based on the conditions in 
which the vehicle enters the road can be appropriately displayed. 

As shown in Figure 3, the image pickup device pertaining to Embodiment 2 for executing 
the present invention is provided with periphery image pickup portion 1 consisting of a plurality 
of cameras that obtain images of the periphery of a vehicle, wherein the camera that obtained the 
images displayed to the driver of the vehicle and the image range of the images obtained by said 
camera that is displayed to the driver of the vehicle are selected in accordance with the 
conditions in which the vehicle enters the road. 

More specifically, the image pickup device shown in Figure 3 provides a plurality of 
cameras 10, 1 1, 17 that take images of the periphery of the vehicle, starting switch 12 that detects 
the starting point at which images picked up by said plurality of cameras 10, 1 1 and 17 begin to 
get displayed to the driver of the vehicle, entry conditions detecting portion 3 that detects the 
conditions in which a vehicle enters the road at the display starting point detected by starting 
switch 12, camera selecting portion 18 that appropriately selects cameras 10, 1 1 and 17 that 
obtain the images displayed to the driver of the vehicle based on the entry conditions detected by 
entry conditions detecting portion 3, image range adjusting portion 15 that adjusts the image 
range displayed based on the entry conditions detected by entry conditions detecting portion 3 
and display monitor 16 that displays the image range that is adjusted by image range adjusting 
portion 15 from images taken by cameras 10, 1 1 and 17 as selected by camera selecting portion 
18. The plurality of cameras that comprise periphery image pickup portion 1 include left wide- 
angle camera 10 and right wide-angle camera 1 1 that are arranged at either side of the front of 
the vehicle and front wide-angle camera 17 arranged on the front of the vehicle. Left wide-angle 
camera 10, right wide-angle camera 1 1 and front wide-angle camera 17 can each obtain images 
at a wide angle of approximately 180 degrees. 

Thus, compared to the image pickup device in Figure 1, the image pickup device shown 
in Figure 3 differs in that it provides additional front wide-angle camera 17 and camera selecting 
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portion 18, but the rest of the constitution is the same as that shown in Figure 1, so further 
explanation is omitted. 

In recent years, image pickup of the periphery of a vehicle obtained by as few cameras as 
possible such as wide-angle cameras 10, 1 1, 1 7 and 20 installed at the front and side of vehicle 
24, as shown in Figure 9, have been considered. In such a case, wide-angle cameras 10 and 11, 
which take images from the side of the vehicle, are installed on the outermost part of the vehicle, 
or the side mirrors, instead of the front of the vehicle, as shown in Figure 8, in order to efficiently 
take images of the periphery of vehicle 24. Left wide-angle camera 10 and right wide-angle 
camera 1 1 obtain images of 180-degree wide range areas 22a and 22b. In the same manner, front 
wide-angle camera 17 and rear wide-angle camera 20, which are arranged at the center front and 
center rear of vehicle 24, respectively, obtain images of 180-degree wide range areas 22c and 
22d. 

For Figure 7(b), the vehicle provided with the image pickup device shown in Figure 3 
and that is disposed with the camera arrangement shown in Figure 9, is shown diagonally 
entering the road from a parked position in the forward-moving direction. In other words, 
traveling direction 21 indicates a diagonal entry condition and not a perpendicular direction in 
relation to the road. 

First, as shown in Figure 10, when the vehicle starts to enter the road from a parking lot 
or a narrow road, or in other words, when only the front end of the vehicle enters the side on 
which the road is located, images of the left and right of the road taken by said front wide-angle 
camera 17 that is installed at the front end of vehicle 24 are cropped and displayed onto display 
monitor 16. 

Then, as shown in Figure 11, when vehicle 24 enters the road and attempts to turn left, 
front wide-angle camera 17 crops only range 23a, which corresponds to the appropriate blind 
spot for confirming the left side from the images taken, and ttien switches over to the images for 
the image range on the right side taken by right wide-angle camera 1 1 installed at the right side, 
crops optimum image range 23b from right wide-angle camera 1 1 and displays it onto display 
monitor 16. 

In this manner, when the vehicle enters the road in the forward-moving direction, first, 
the image taken by front wide-angle camera 17 is displayed and when an image of one side of 
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the road cannot be picked up by front wide-angle camera 17, the images taken by side cameras 
10 and 1 1 are displayed. As the vehicle proceeds further onto the road, the image range of the 
right side that could not be obtained by front wide-angle camera 17 can be displayed via camera 
11, which is installed on the right side, so the system can switch to the optimum camera for 
displaying the image of the picture angle range of the side view, depending on the entry 
conditions. However, it is necessary to choose the camera that covers the necessary picture angle 
range based on the entry conditions of vehicle 24. 

When the angle of entry in relation to the road is at an angle to begin with, images of the 
left and right directions of the road can be displayed by properly selecting the range to be 
photographed and the camera in accordance with the angle of the vehicle at the point of entry. 
The correct and optimum display can be performed by pre-storing the relation between the image 
range obtained and the camera that is selected when switching the image range in relation to the 
road situation calculated by navigation 13 and gyro 14, as was done in Embodiment 1, 

Next is provided an explanation of the process for the image pickup device shown in 
Figure 3, with reference to the flow chart shown in Figure 4. 

(a) First at Step S201, the ignition is turned ON at the discretion of the driver. At Step 
S202, the process determines whether the ignition has been turned OFF or not by the driver. If 
the ignition is ON (is ON at Step S202), the process proceeds to 8203 and if the ignition is OFF 
(is OFF at Step S202), the flow process ends. 

(b) Next, at Step S203, the process determines the start of the image pickup device shown 
in Figure 3. If the image pickup device has been started (if it is ON at Step S203), the process 
moves to Step S204 and if the image pickup device has not been started (if it is OFF at Step 
S203), the process returns to Step S202. 

(c) Next, at Step S204, camera selecting portion 1 8 obtains the image signal forwarded 
from wide-angle cameras 10, 1 1 and 17 and at Step S205, camera selecting portion 18 obtains 
the current position of the vehicle and the map information of the vicinity from navigation 13. At 
Step S206, camera selecting portion 18 obtains the absolute direction information of the vehicle 
from gyro 14. 

(d) Next, at Step S207, the current position of the vehicle and the map information of the 
vicinity from navigation 13 and the absolute direction information of the vehicle from gyro 14 
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are used to calculate the direction of entry onto the road (forward or reverse direction/angle of 
entry (the angle in relation to the road)/entry position (the distance at which the vehicle advances 
onto the road). At Step S208, the position (absolute position)/direction (absolute direction) of the 
camera installed on the vehicle is specified. 

(e) Next, at Step S209, the range required for observation is specified in accordance with 
the entry conditions and at Step S210, camera selecting portion 18 selects camera 10, 1 1 or 17 
that picks up the image range required for observation in accordance with the entry conditions. 
At Step S21 1 image range adjusting portion 15 adjusts the image range required for observation 
from the images taken by the selected camera 10, 1 1 and 17. Finally, at Step S212, display 
monitor 16 displays the adjusted image to the driver. After this, the process returns to S202 and 
Steps S202'-S212 are repeated to perform the process again. 

It is desirable to execute Steps S202'-S212 while the vehicle is in the process of entering 
the road as well as after the vehicle has entered the road. In other words, even after the vehicle 
has entered the road, it is desirable to select cameras 10, 1 1 or 17 which have obtained the 
images displayed to the driver of the vehicle and the image range of the images obtained by 
cameras 10, 11 or 17 that is displayed to the driver of the vehicle. 

As explained above, since the image pickup device pertaining to Embodiment 2 is 
provided with entry conditions detecting portion 3 that detects the conditions in which a vehicle 
enters the road at the display starting point that is displayed to the driver and image range 
adjusting portion 15 that adjusts the image range displayed in accordance with the conditions in 
which the vehicle enters the road, the optimum range of observation based on the conditions in 
which the vehicle enters the road can be appropriately displayed. 

In addition, since the image pickup device pertaining to Embodiment 2 is provided with 
camera selecting portion 18 for appropriately selecting camera 10, 1 1 or 17 based on the 
conditions in which the vehicle enters the road, it can appropriately display the optimum range of 
observation for any type of entry conditions. 

As shown in Figure 5, the image pickup device pertaining to Embodiment 3 of the 
present invention provides periphery image pickup portion 1 comprised of a plurality of cameras 
that obtain images of the periphery of a vehicle and the camera that obtains the images displayed 
to the driver of the vehicle and the image range of the images obtained by said camera that is 
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displayed to the driver of the vehicle are selected in accordance with the conditions in which the 
vehicle enters the road. 

More specifically, the image pickup device shown in Figure 5 provides a plurality of 
cameras 10, 1 1 and 20 that take images of the periphery of a vehicle, starting switch 12 that 
detects the starting point at which images picked up by said plurality of cameras 10, 1 1 and 20 
begin to get displayed to the driver of the vehicle, entry conditions detecting portion 3 that 
detects the conditions in which a vehicle enters the road at the display starting point detected by 
starting switch 12, camera selecting portion 18 that appropriately selects cameras 10, 1 1 and 20 
that obtain the images displayed to the driver of the vehicle based on the entry conditions 
detected by entry conditions detecting portion 3, image range adjusting portion 1 5 that adjusts 
the image range displayed based on the entry conditions detected by entry conditions detecting 
portion 3, image synthesizing portion 19 that synthesizes the images taken by the plurality of 
cameras, and display monitor 16 that displays the synthesized image range that is adjusted by 
image range adjusting portion 15 from images taken by cameras 10, 1 1 and 20 as selected by 
camera selecting portion 18. The plurality of cameras that comprise periphery image pickup 
portion 1 include left wide-angle camera 10 and right wide-angle camera 1 1 that are arranged at 
either side of vehicle 24 and rear wide-angle camera 20 arranged at the rear of the vehicle 24. 
Left wide-angle camera 10, right wide-angle camera 1 1 and rear wide-angle camera 20 can each 
obtain images at a wide angle of approximately 1 80 degrees. When a plurality of images taken 
by the plurality of cameras 10, 1 1 and 20 include an overlapping portion in the image range 
adjusted by image range adjusting portion 15, image synthesizing portion 19 synthesizes said 
plurality of images. In other words, when the image range adjusted by image range adjusting 
portion 15 extends into plurality of cameras 10, 1 1 and 20, image synthesizing portion 19 
synthesizes the images. 

Thus, compared to the image pickup device in Figure 1, the image pickup device shown 
in Figure 5 differs in that it provides additional rear wide-angle camera 20, camera selecting 
portion 18 and image synthesizing portion 19, but the rest of the constitution is the same as that 
shown in Figure 1, so further explanation is omitted. 

Figure 7(c) shows a vehicle equipped with the image pickup device shown in Figure 5 
and that has the camera arrangement shown in Figure 9 and is an illustration of a diagonal entry 
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onto the road in the reverse direction from the side on which the parking lot is located to the side 
on which the road is located. In other words, traveling direction 21 indicates a diagonal entry 
condition and not a perpendicular direction in relation to the road. 

First, when the vehicle starts to enter the road from a parking lot or a narrow road, or in 
other words, when only the front end of the vehicle enters the side on which the road is located, 
an image of the left and right of the road taken by rear wide-angle camera 20 that is installed at 
the rear of vehicle 24 is cropped and displayed onto display monitor 1 6. 

Then, as shown in Figure 7(c), as vehicle 24 proceeds even further onto the road, image 
range 22d taken by rear wide-angle camera 20 and left side image range 22a taken by left wide- 
angle camera 10 overlap. When the camera that covers the necessary picture angle range based 
on the entry conditions of the vehicle is rear wide-angle camera 20 and left wide-angle camera 
10, image ranges 22d and 22a taken by rear wide-angle camera 20 and left wide-angle camera 10 
are synthesized and displayed on display monitor 16. In other words, when vehicle 24 enters the 
road in the reverse direction, the image taken by left wide-angle camera 10 and the image taken 
by rear wide-angle camera 20 are synthesized and displayed. 

In this manner, as shown in Figure 7(c) for example, when entering the road slowly in the 
reverse direction or when entering a parking lot from the road in the reverse direction or the like, 
the image ranges from the camera on the side of the vehicle and the camera on the rear of the 
vehicle are appropriately changed in accordance with the conditions in which the vehicle enters 
the road, are synthesized into an image with a 1 80 degree range of the rear of the vehicle and 
displayed. By synthesizing an image of the road into a 180-degree range and displaying it, the 
driver can easily confirm obstacles on the road to the rear periphery of the vehicle. It also 
becomes easier to confirm obstacles on a sidewalk when entering the road from the sidewalk. 

When the angle of entry in relation to the road is at an angle to begin with, the image of 
the rear periphery of the vehicle can be displayed by appropriately selecting the range to be taken 
and the camera based on the angle of the vehicle at the point of entry. By switching the image 
range and the camera that obtains the images, the correct and optimum display can be performed 
by pre-storing the relation between the image range obtained and the camera that is selected 
when switching in relation to the road situation calculated by navigation 13 and gyro 14, as was 
done in Embodiment 1. 
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Next, an explanation is provided of the process for the image pickup device shown in 
Figure 5 with reference to the flow chart in Figure 6. 

(a) First at Step S301, the ignition is turned ON at the discretion of the driver. At Step 
S3 02, the process determines whether the ignition has been tumed OFF or not by the driver. If 
the ignition is ON (is ON at Step S302), the process proceeds to S3 03 and if the ignition is OFF 
(is OFF at Step S302), the flow process ends. 

(b) Next, at Step S303, the process determines the start of the image pickup device shown 
in Figure 5. If the image pickup device has been started (if it is ON at Step S303), the process 
moves to Step S304 and if the image pickup device has not been started (if it is OFF at Step 
S303), the process returns to Step S302. 

(c) Next, at Step S3 04, camera selecting portion 18 obtains the image signal forwarded 
from wide-angle cameras 10, 1 1 and 20 and at Step S305, camera selecting portion 18 obtains 
the current position of the vehicle and the map information of the vicinity from navigation 13. At 
Step S306, camera selecting portion 18 obtains the absolute direction information of the vehicle 
from gyro 14. 

(d) Next, at Step S307, the current position of the vehicle and the map information of the 
vicinity from navigation 13 and the absolute direction information of the vehicle from gyro 14 
are used to calculate the direction of entry onto the road (forward or backward direction/angle of 
entry (the angle in relation to the road)/entry position (the distance at which the vehicle advances 
onto the road). At Step S308, the position (absolute position)/direction (absolute direction) of the 
camera installed on the vehicle is specified. 

(e) Next, at Step S309, the range required for observation is specified in accordance with 
the entry conditions and at Step S3 10, camera selecting portion 18 selects camera 10, 11 or 20 
that takes the required observation range based on the entry conditions. At Step S31 1, image 
range adjusting portion IS adjusts the images taken by selected cameras 10, 1 1 and 20 to the 
image range required for observation. At Step S3 12, image synthesizing portion 19 synthesizes 
the plurality of images taken by camera 10, 1 1 and 20. Finally, at Step 313, display monitor 16 
displays the synthesized image to the driver. After this, the process retums to S302 and Steps 
S302~S3 13 are repeated to perform the process again. 
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It is desirable to execute Steps S302-'S313 while the vehicle is in the process of entering 
the road as well as after the vehicle has entered the road. In other words, even after the vehicle 
has entered the road, it is desirable to select cameras 10, 1 1 and 20, which have obtained the 
images displayed to the driver of the vehicle and the image range of the images obtained by 
cameras 10, 1 1 and 20 that is displayed to the driver of the vehicle and synthesize these plurality 
of images. 

As explained above, since the image pickup device pertaining to Embodiment 3 for 
executing the present invention is provided with entry conditions detecting portion 3 that detects 
the conditions in which a vehicle enters the road at the point at which images start getting 
displayed to the driver and image range adjusting portion 1 5 that adjusts the image range 
displayed in accordance with the conditions in which the vehicle enters the road, the optimum 
range of observation based on the conditions in which the vehicle enters the road can be 
appropriately displayed. 

In addition, since the image pickup device pertaining to Embodiment 3 is provided with 
camera selecting portion 18 for appropriately selecting camera 10, 11 and 20 based on the 
conditions in which the vehicle enters the road, it can appropriately display the optimum range of 
observation for any type of entry conditions. 

As shown in Figure 14, the image pickup device pertaining to Embodiment 4 is provided 
with a plurality of periphery image pickups portion 200 (periphery image pickup means) for 
obtaining image of the periphery of a vehicle wherein periphery image pickups portion 200 for 
obtaining images to be displayed to the driver of the vehicle and the image range of the images 
obtained by periphery image pickups portion 200 that are displayed to the driver of the vehicle 
are selected and synthesized. 

More specifically, it is provided with plurality of periphery image pickups portion 200 
consisting of plurality of cameras 101'-106 for taking images of the periphery of a vehicle, 
vehicle position acquiring portion 201 (vehicle position acquiring means) for acquiring the 
global position of a vehicle, vehicle direction acquiring portion 202 (vehicle direction acquiring 
means) for acquiring the global direction of a vehicle, road information acquiring portion 203 
(road information acquiring means) for acquiring road information for the periphery of a vehicle, 
image selecting portion 204 (image selecting means) for selecting the images from cameras 
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101-106 to be used, based on the position of each camera on the vehicle, the vehicle position, 
the vehicle direction and road information and display monitor 206 (display means) for 
displaying the selected image or a plurality of images to the driver of the vehicle. 

Vehicle position acquiring portion 201 acquires the global position of the vehicle. 
Vehicle direction acquiring portion 202 acquires the global direction of the vehicle. Road 
information acquiring portion 203 acquires road information for the periphery of the vehicle. 
Since devices 201 through 203 are provided to function as the navigation system, this 
information can be acquired from a navigation system. 

The information for the global position of the vehicle, the global direction of the vehicle 
and the road information acquired by devices 201'-203 is sent to image selecting portion 204. 
Inside of image selecting portion 204 is stored information on the image pickup range and the 
direction and position of cameras 101-- 106 installed on the vehicle. The speed of the vehicle can 
be calculated from the global position information and the global direction information of the 
vehicle, or can, of course, be obtained directly from the vehicle. 

Image synthesizing portion 205 selects the images from the camera[s] to be used based 
on a list of cameras 101-106 selected by image selecting portion 204 and synthesizes the 
pictures arranged on the display screen. Commonly known technology pertaining to the field of 
image processing can be used for the actual picture synthesizing operation. 

Cameras 101-106 in Figure 14 are installed in the position on vehicle 24, as shown in 
Figure 1 5(a), for example. Each of cameras 102-106 can take pictures of image pickup ranges 
107-1 11, respectively. Here, the direction of the vehicle (traveling direction 21) is set in the 
direction shown in Figure 15(b). Vehicle direction 21 is set at 0 degrees, the rear direction at 180 
degrees, the right direction at 90 degrees and the left direction at -90 degrees. This direction 
information is used to express image pickup ranges 107-1 11. Specifically, 0 degrees, 180 
degrees, 90 degrees and -90 degrees are the base angles and each of image pickup ranges 
107-1 1 1 is defined as the starting angle: passage angle: ending angle. The passage angle is any 
one of the base angles. Therefore, image pickup range 107 is expressed as -30:-90:-120, image 
pickup range 108 is expressed as 30:90:120, image pickup range 109 is expressed as - 
1 20: 1 80: 1 80, image pickup range 1 1 0 is expressed as 1 20: 1 80: 1 80 and image pickup range 1 1 1 
is expressed as -90:180:90. 



234.604WO01 



18 

The starting angle is set at the side of traveling direction 21 (front side), for example. For 
this setting, if the starting angle is in the same position as the front side of the vehicle (image 
range 1 1 1, for example), the starting position is set as the left side in relation to traveling 
direction 21 of the vehicle. 

Figures 16-- 18 are explanatory diagrams of various situations in which the image pickup 
device shown in Figure 14 operates. Element (a) in each Figure is a road diagram showing the 
position of vehicle 24 and road 210 and (b) is a type diagram showing base line A and target 
range B set by the image pickup device. The direction in which the acute angle of the isosceles 
triangle indicated as vehicle 24 is pointed is the traveling direction of vehicle 24. 

Image pickup portion 204 in Figure 14 corrects the relationship between vehicle 24 and 
road 210 that intersects with the traveling distance of vehicle 24 to base line A and target range B 
of vehicle 24, as shown in Figure 16(b), for example, based on the road information, the global 
position and global direction of vehicle 24 and the vehicle speed. 

Figures 16 and 17 are examples of vehicle 24 moving in the forward direction and 
merging with another road 210a and 210b and Figure 18 is an example of vehicle 24 moving in 
the reverse direction and merging with another road 210c. In either case, for Figures 16 18, the 
vehicle draws base line A as it approaches road 210 onto which it is about to merge. 

Next, the range that the driver wants to view is set as target range B. Target range B can 
simply be set as a 180-degree angle at the side on which vehicle 24 is not located in relation to 
base line A, as shown in Figures 16(b) and 18(b), for example. Or, the setting can be changed in 
accordance with the situation of the road that the vehicle is about to merge with. For example, 
for the situation shown in Figure 17(a), since right tums are prohibited for road 210b, onto which 
the vehicle is about to merge, so when vehicle 24 knows that it can only proceed in the left 
direction based on road information from road information acquiring portion 203, target range B 
is set on the right side of road 210b onto which the vehicle is about to merge, as shown in Figure 
17(b). 

As was the case with image pickup ranges 107-1 1 1 for each camera, as shown in Figure 
15(a), target range B also has vehicle direction (traveling direction) 21 as its base and is defined 
according to three angles (starting angle: passage angle: ending angle). So, for example, for the 
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situation in Figure 16, target range B is defined as -90:0:90 and as 45:90:120 for the situation 
shown in Figure 17 and -60:180:120 for the situation shown in Figure 18, 

Thus, image selecting portion 204, shown in Figure 14, selects images fi-om cameras 

101- 106 based on target range B, which is determined from the relationship between vehicle 24 
and road 210, and image pickup ranges 107-1 1 1 from each of cameras 101-106 on vehicle 24. 

Next is provided an explanation of the method used to prioritize cameras 101-106 when 
image selecting portion 204, shown in Figure 14, selects images from cameras 101-106, with 
reference to Figure 19, Figure 19 corresponds to the situation shown in Figure 18. 

(a) For each situation in Figures 16-18, image selecting portion 204 obtains the straight- 
line distances, C2^C^, from each of cameras 102-106 on vehicle 24 to the road (base line A) and 
prioritizes each of cameras 102-106. Straight-line distances C2'^C6 are prioritized so that the 
camera at the shortest distance has the highest priority. For example, if straight-line distances 
C2--C6 to base line A are considered to be those shown in Figure 19, then the priority of cameras 

102- 106, in the order of highest priority, would be 106, 104, 105, 102 and 103. When the 
positions of each of cameras 102-106 move beyond base line A to the side on which target range 
B has been set, or in other words, when a camera enters road 210, the priority of that camera 
should be set so that it is higher than that of the cameras that have not moved beyond base line 
A. 

(b) Next, image selecting portion 204 uses image pickup ranges 107-1 1 1 from each of 
cameras 102-106 and the priority of each of cameras 102-106 to decide which camera to use. 
Specifically, as explained above, target range B in Figure 18(b) is -60:180:120. The cameras are 
considered in order of highest priority to decide which camera to use. For example, the camera 
with the highest priority is camera 106. Image pickup range 1 1 1 of camera 106 is -90:180:90. 
Since there is an overlapping range (-90:180:120) when target range B, shown in Figure 18(b), 
and image pickup range 1 1 1 of camera 106 are compared, camera 106 is selected as the camera 
to be used. 

(c) Next, overlapping range (-90:180:120) is excluded fi"om target range B (-60:180:120). 
And as a result, the remaining range of target range B is -60:-90:-90. If there is no overlapping 
range between target range B and the image pickup range of the camera with the highest priority, 
the camera with the highest priority is not selected so there is no range to be excluded. In the 
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example shown in Figure 19, since there is a remaining range for target range B, the same 
process is performed for the camera with the second highest priority, or camera 104. Image 
pickup range 109 of camera 104 is -120:180:180. And since there is no overlapping range 
compared to the remaining range of target range B (-60:-90:-90), camera 104 is not selected as 
the camera to be used. 

(d) The same process is then performed for camera 105, which has the third highest 
priority, since there is still a remaining range in target range B. However, since image pickup 
range 1 10 of camera 105 also has no overlapping range with the remaining range (-60:-90:-90) of 
target range B, it is not selected as the camera to be used. Then, when the same process is 
performed for camera 102, which has the fourth highest priority, since there is a portion that 
overlaps with the remaining range of target range B, image selecting portion 204 selects camera 
102 as the camera to be used. Then, when the range that overlaps with image pickup range 107 
of camera 102 is excluded from the remaining portion of target range B, the remaining portion 
disappears. At the point at which the remainder of target range B disappears, the cameras with 
lower priorities are not used. In this example, it is decided not to use camera 103 without 
making any comparisons. According to the aforementioned process, cameras 106 and 102 are 
selected as the cameras to be used. Image selecting portion 204 sends a list of the cameras 
selected according to the aforementioned process to image synthesizing portion 205. 

In this manner, image selecting portion 204 selects images from cameras 101--106 based 
on straight-line distances Ci'^e, target range B and image pickup ranges 107-1 1 1 from each of 
the cameras on vehicle 24. It can then be decided which camera to use from cameras 101-106 by 
conducting a simple comparison and a faster, less expensive image pickup device can be 
provided. 

Cameras 102 and 103 are selected as the cameras to be used in the same manner for the 
situation shown in Figure 16. In this case, when the aforementioned process is performed for all 
of the cameras, there will be a range remaining in image range B shown in Figure 16, but this 
range can be designated as a range in which image pickup cannot be performed. 

In addition, camera 103 is selected as the camera to be used in the same manner for the 
situation in Figure 17. Although three or more cameras can be simultaneously selected 
depending on the arrangement of the cameras and the image pickup range, it is not desirable for 
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the driver to view a display in which too many images are displayed at once, so selection can be 
restricted to the first two cameras that are selected for use. 

As shown in Figure 20, display monitor 206 shown in Figure 14 is equipped with display 
screen 220 that consists of two independent display areas, (left) display area 221 and (right) 
display area 222 and can simultaneously display images fi:om two cameras selected by image 
selecting portion 204. When display monitor 206 displays the images selected by image selecting 
portion 204, it determines the positional relationship for the display with consideration made to 
the positional relationship of the original camera. 

Image synthesizing portion 205 selects the image from the camera being used based on 
the list of cameras being used sent from image selecting portion 204 and synthesizes the pictures 
arranged on display screen 220 of display monitor 206. Commonly known technology pertaining 
to the field of image processing can be used for the actual picture synthesizing operation. When 
this takes place, the positional relationship of the mutual images reflects the physical positional 
relationship of the original camera. Specifically speaking, the position is first determined 
according to the positional relationship shown in the "X" direction in Figure 21 . At this point, the 
left/right relationship is not changed. Then when the positions in the "X" direction are the same, 
the image from the camera for which the "Y" direction is the minus direction is placed outside of 
display monitor 206. 

If the situation shown in Figure 18 is used as an example, the image from camera 102 is 
displayed in display area (left) 221 shown in Figure 20. If the situation shown in Figure 17 is 
used as an example, the image from camera 103 is displayed in display area (left) 221 and the 
image from camera 105 is displayed in display area (right) 222. If the situation shown in Figure 
16 is used as an example, the image from camera 102 is displayed in display area (left) 221 and 
the image from camera 103 is displayed in display area (right) 222. 

The synthesized image created according to the aforementioned process is sent to display 
monitor 206 and displayed to the driver. 

Next is provided an explanation of the image pickup method employed by the image 
pickup device shown in Figure 14, with reference to Figure 22. 

(a) First, at Step S401, images of the periphery of vehicle 24 are taken by plurality of 
cameras 1 01-- 106. 
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(b) Then, the process proceeds to Step S402 and vehicle position acquiring portion 201 
acquires the global position of vehicle 24. Next, the process proceeds to Step S403 and vehicle 
direction acquiring portion 202 acquires the global direction of the vehicle. 

(c) Next, road information for the vicinity of vehicle 24 is acquired by road information 
acquiring portion 203. The process proceeds to Step S404 and image selecting portion 204 
selects the images from camera 106, which is the camera to be used, from the positions of each 
of cameras 101-106 on vehicle 24, the global position, the global direction and the road 
infomiation. 

(d) The process then proceeds to Step S405 and the image or plurality of images selected 
is displayed to the driver of vehicle 24 using display monitor 206. Thus, according to the 
aforementioned process, the image pickup device shown in Figure 14 can select the camera 
image to display and display it to the driver based on the circumstances of the periphery of 
vehicle 24. 

The following effects can be achieved by Embodiment 4 for the present invention, as 
explained above. 

Image selecting portion 204 selects images from cameras 101-- 106 that are being used 
based on road information acquired by road information acquiring portion 203 and vehicle 
information (global position and global direction) acquired by vehicle position acquiring portion 
201 and vehicle direction acquiring portion 202 so that only the range needed by the driver can 
be displayed on display monitor 206 in order to make it easier for the driver to view images. 

Image selecting portion 204 selects images from cameras 101-'106 that are being used 
based on straight-line distances C2-C6 between each of cameras 101-106 on vehicle 24 and road 
210, image range B that is determined from the relationship between vehicle 24 and road 210, 
and image pickup ranges 107-1 1 1 of cameras 101-106 on vehicle 24 so that selection of 
cameras 101-106 that are being used can be performed by means of a simple comparison in 
order to provide a faster, less expensive image pickup device. 

The priority for each of cameras 102-106 is set so that the camera with the shortest 
distance for straight-line distances C2'-C6 between each of cameras 102-106 on vehicle 24 and 
road 210 has the highest priority in order to realize this function by means of a simpler algorithm 
and provide a faster, less expensive image pickup device. 
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Target range B and pickup image ranges 1 07^-1 1 1 are represented only as angles so that 
the comparison can be carried out by means of a simpler algorithm in order to provide a faster, 
less expensive image pickup device. 

When display monitor 206 displays the images selected by image selecting portion 204, it 
determines the positional relationship for the display with consideration made to the positional 
relationship of original cameras 101-106 to allow for an easier-to-view display and improve the 
viewing for the driver. 

Conventionally, for the device in which multiple cameras 101^-106 installed on vehicle 
24 display images to the driver, a switch would have to be operated by the driver to switch the 
camera images so that multiple images could be displayed at once. However, such a device had 
problems because the switching operation was troublesome and unwanted images would get 
displayed. However, according to Embodiment 4, a device is provided that can automatically 
select the images from the relationship between vehicle 24 and road 210 and display them to the 
driver. 

As explained above, the present invention has been described according to Embodiments 
1 though 4 but it should not be interpreted that constitution of a portion of the disclosure or that 
the Figures limit the present invention. Based on the present disclosure, it is obvious to a person 
skilled in the art that there are various other embodiments that could be substituted to enforce the 
present invention as well as other working examples and operating technologies. In other words, 
it should be interpreted that the present invention encompasses various different types of 
embodiments that have not been described herein. Therefore, based on the present disclosure, the 
present invention is only limited to specific items of the invention pertaining to the appropriate 
claims of the invention. 

Although specific embodiments have been illustrated and described herein, it will be 
appreciated by those of ordinary skill in the art that any arrangement, which is calculated to 
achieve the same purpose, may be substituted for the specific embodiment shown. This 
application is intended to cover any adaptations or variations of the present invention. Therefore, 
it is manifestly intended that this invention be limited only by the claims and the equivalents 
thereof 



